Featurizing Text: Converting Text into Predictors for Regression Analysis

Modern data streams routinely combine text with the familiar numerical data used in regression analysis. For example, listings for real estate that show the price of a property typically include a verbal description. Some descriptions include numerical data, such as the number of rooms or the size of the home. Many others, however, only verbally describe the property, often using an idiosyncratic vernacular. For modeling such data, we describe several methods that that convert such text into numerical features suitable for regression analysis. The proposed featurizing techniques create regressors directly from text, requiring minimal user input. The techniques range naive to subtle. One can simply use raw counts of words, obtain principal components from these counts, or build regressors from counts of adjacent words. Our example that models real estate prices illustrates the surprising success of these methods. To partially explain this success, we offer a motivating probabilistic model. Because the derived regressors are difficult to interpret, we further show how the presence of partial quantitative features extracted from text can elucidate the structure of a model. Key Phrases: sentiment analysis, n-gram, latent semantic analysis, text mining ∗Research supported by NSF grant 1106743